Adapt openstack_test stage to OTE (drop legacy make openstack-tests) by tusharjadhav3302 · Pull Request #33 · shiftstack/shiftstack-qa

tusharjadhav3302 · 2026-08-07T11:48:37Z

Summary

After openshift/openstack-test#303 (OSASINFRA-4375, merged 2026-08-04) dropped the legacy make openstack-tests target on release-4.20+, shiftstack-qa’s openstack_test stage fails immediately with No rule to make target 'openstack-tests'.
For discovered_openshift_release >= 4.20, build with make extension, install bin/openstack-test-tests-ext, list via list --suite openstack-test/all -o names, and run via run-suite (or run-test + stdin when allow/block lists are set).
Keep the legacy make openstack-tests / run openshift/openstack path for < 4.20 so existing 4.16/4.17 job defs still work.

CLI / behavior trade-offs

Legacy	OTE (>= 4.20)
`make openstack-tests`	`make extension`
`run openshift/openstack --dry-run`	`list --suite … -o names`
`run … -f list --junit-dir`	Unfiltered: `run-suite … --junit-path=junit_e2e_openstack.xml`; filtered: `run-test` via stdin (OTE has no `-f` / `--junit-path` on `run-test`) + synthesize junit XML from JSON for `post_openshift_tests`
Log lines `passed:` / `failed:`	JSON `"result"` field; rescue counters updated accordingly

Env vars unchanged: OS_CLOUD, KUBECONFIG, RHOSO_KUBECONFIG, SHIFTSTACK_PASS_FILE.

Test plan

On shiftstackclient (serval70, guest OCP 4.23.0-0.nightly-2026-08-05-200523): clone release-4.23, confirm make openstack-tests fails and make extension + list --suite openstack-test/all -o names succeeds.
Run one real test via OTE run-test against the live guest cluster (bz_2022627) — PASSED ("result": "passed"). See PR comment for full evidence.
Full Ansible stage-only run against an existing cluster (stages: [openstack_test]) — blocked in this env by undefined hypervisor (cifmw-dnsmasq delegate); OTE CLI path above still validates the migration.
Optional: exercise allowlist path via lb_tests (filtered run-test + synthesized junit).
Note: verification-only jobs that omit openstack_test will not exercise this change.

Links

Upstream: OSASINFRA-4375: Drop Legacy openshift-test entry point: releases 4.23+ openshift/openstack-test#303 (backports: #306, #307, #308)
Jira: https://redhat.atlassian.net/browse/OSASINFRA-4375

tusharjadhav3302 · 2026-08-07T12:31:45Z

Manual validation evidence (serval70 / shiftstackclient)

Env

Host: serval70 → oc rsh -n openstack shiftstackclient-shiftstack
Guest OCP: 4.23.0-0.nightly-2026-08-05-200523 (AVAILABLE)
Clone: ~/artifacts/openstack-test-repro @ release-4.23

1. Legacy path broken (root cause reproduced)

make openstack-tests
make: *** No rule to make target 'openstack-tests'.  Stop.

2. OTE build succeeds

$ make extension
Building OTE test extension binary...
GOTOOLCHAIN=auto GOSUMDB=sum.golang.org go build -mod=vendor -o bin/openstack-test-tests-ext ./cmd/extension
go: downloading go1.23.4 (linux/amd64)
Extension binary built: bin/openstack-test-tests-ext

$ test -x bin/openstack-test-tests-ext && echo "binary OK"
binary OK

3. List suite (replaces `--dry-run`)

$ ./bin/openstack-test-tests-ext list --suite openstack-test/all -o names | head
[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2022627: Machine should report all openstack instance addresses
[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2073398: [Serial] MachineSet scale-in does not leak OpenStack ports
...

4. Run one real test against the live cluster — PASSED

$ ./bin/openstack-test-tests-ext run-test \
  '[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2022627: Machine should report all openstack instance addresses'

Will run 1 of 1 specs
...
Ran 1 of 1 Specs in 6.713 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[
  {
    "name": "[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2022627: Machine should report all openstack instance addresses",
    "result": "passed",
    "duration": 6713,
    ...
  }
]

Notes

This validates the OTE build/list/run path that this PR wires into the openstack_test role for discovered_openshift_release >= 4.20.
A full ansible-navigator stage-only run was attempted but failed early on hypervisor undefined (cifmw-dnsmasq delegate_to); ~/artifacts/openstack-test-results/ stayed empty because the suite never started. Re-run with --extra-vars '{"hypervisor":"serval70",...}' if full role wiring needs a check later.

danchild · 2026-08-07T12:36:33Z

Thanks for catching this @tusharjadhav3302

For documentation purposes, the behavior you're seeing is due to these PR's as well:

openshift/openstack-test#308
openshift/openstack-test#307
openshift/openstack-test#306

tusharjadhav3302 · 2026-08-08T18:38:18Z

@danchild Done.

imatza-rh

/lgtm

Upstream openstack-test removed make openstack-tests on release-4.20+ (PR #303 and backports). Use make extension / run-suite for those releases, keep the legacy path for older branches, and preserve run-test/junit exit codes for filtered runs. Co-authored-by: Cursor <cursoragent@cursor.com>

openshift-ci · 2026-08-09T16:04:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from imatza-rh. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Outer JSON marks every result failed when stderr starts with klog "I..." lines. Count and rewrite junit from nested STDOUT / ginkgo SUCCESS|FAIL so real suite failures stay UNSTABLE instead of "no tests run". Co-authored-by: Cursor <cursoragent@cursor.com>

tusharjadhav3302 · 2026-08-10T06:38:44Z

Follow-up: OTE false-failure counting (pushed)

run-suite was marking all outer JSON results as `failed` due to `Deserializaion Error: invalid character 'I'` (klog on stderr). That made rescue counters 0/0 and hard-failed with "no tests run", and junit showed 51 failures.

Fix (this PR): `files/ote_resolve_results.py` resolves true outcomes from nested STDOUT JSON / ginkgo `SUCCESS!`|`FAIL!`, used to:

count passed/failed in rescue
rewrite `junit_e2e_openstack.xml` after `run-suite` / filtered `run-test`

Validated against the 4.23 TP log: 24 passed / 20 skipped / 7 real failed (not 51).

Skipped Ansible tasks still overwrite a shared register, which cleared OTE resolved counts and falsely triggered "no tests where run!". Co-authored-by: Cursor <cursoragent@cursor.com>

tusharjadhav3302 · 2026-08-10T07:12:51Z

Pushed register fix: skipped legacy count tasks no longer wipe OTE resolved pass/fail counts (5781579).

tusharjadhav3302 · 2026-08-10T07:41:40Z

Manual validation (serval71 / live RHOSO + guest OCP 4.23)

Validated on shiftstackclient against an existing cluster (no full TP recheck yet). Collection installed from this PR / combine #34 tip including 5781579 (register-count fix).

What we validated

OTE path — make extension, run-suite openstack-test/all, junit rewrite via files/ote_resolve_results.py
False “51 failed” / “no tests where run!” — outer OTE still reports Error: 51 tests failed (klog deserialize noise), but rescue now resolves real counts and does not hard-fail
Register wipe — distinct OTE vs legacy registers + normalize set_fact; skipped legacy count tasks no longer clear OTE counts
Stage behavior — suite failure → UNSTABLE → must-gather + post (XML/HTML) → dnsmasq restore → play ends failed=0, rescued=1

Evidence (manual `stages: ["openstack_test"]`)

Hypervisor dnsmasq delegate to serval71.lab.eng.tlv2.redhat.com OK
Resolver offline on prior log: passed=24 failed=7 skipped=20
Live suite run printed: openstack-test counts: passed=20, failed=11
Fail task “no tests where run!” was skipped
PLAY RECAP: ok=74 … failed=0 … rescued=1

The ~11 real Amphora/PVC (etc.) failures are suite/product noise on this env; they are correctly treated as UNSTABLE, not as an automation hard fail.

Not claimed

Full osp_verification green / zero openstack-test failures
TP !2584 recheck (pending when ready; pin is DNM: TP combine #32 + #33 #34)

tusharjadhav3302 · 2026-08-10T07:42:54Z

Manual validation (serval71 / live RHOSO + guest OCP 4.23)

Validated on shiftstackclient against an existing cluster (no full TP recheck yet). Collection installed from this PR / combine #34 tip including 5781579 (register-count fix).

What we validated

OTE path — make extension, run-suite openstack-test/all, junit rewrite via files/ote_resolve_results.py
False “51 failed” / “no tests where run!” — outer OTE still reports Error: 51 tests failed (klog deserialize noise), but rescue now resolves real counts and does not hard-fail
Register wipe — distinct OTE vs legacy registers + normalize set_fact; skipped legacy count tasks no longer clear OTE counts
Stage behavior — suite failure → UNSTABLE → must-gather + post (XML/HTML) → dnsmasq restore → play ends failed=0, rescued=1

Evidence (manual `stages: ["openstack_test"]`)

Hypervisor dnsmasq delegate to serval71.lab.eng.tlv2.redhat.com OK
Resolver offline on prior log: passed=24 failed=7 skipped=20
Live suite run printed: openstack-test counts: passed=20, failed=11
Fail task “no tests where run!” was skipped
PLAY RECAP: ok=74 … failed=0 … rescued=1

The ~11 real Amphora/PVC (etc.) failures are suite/product noise on this env; they are correctly treated as UNSTABLE, not as an automation hard fail.

Not claimed

Full osp_verification green / zero openstack-test failures
TP !2584 recheck (pending when ready; pin is DNM: TP combine #32 + #33 #34)

OTE list -o names emits unquoted names plus klog on stdout, while allowlists are converted as quoted patterns for legacy dry-run. Strip optional quotes, drop klog lines, and filter list stdout so .*[lb].* matches again. Co-authored-by: Cursor <cursoragent@cursor.com>

tusharjadhav3302 · 2026-08-10T08:03:27Z

Follow-up: lb_tests / egressip allowlist vs OTE list

Manual `lb_tests` on serval71 failed with allowlist `".[lb]."` not found in `openstack_tests.txt`.

Cause: OTE `list -o names` writes unquoted names (and a leading klog `I0810 ...` line on stdout). Allowlist conversion still emits quoted patterns meant for legacy dry-run, so `filter_tests_list` fullmatch failed. `sed` on the artifact did not help because the playbook regenerates `allowlist.txt` each run.

Fix (this PR):

`filter_tests_list`: strip optional surrounding quotes; ignore klog lines
OTE list task: drop klog from captured names (`grep -v` / stderr redirect)

Re-fetch/reinstall collection before re-running `lb_tests` / `egressip_tests`.

imatza-rh · 2026-08-10T08:26:10Z

/lgtm

tusharjadhav3302 · 2026-08-10T10:22:59Z

Manual validation complete (serval71) — still targeting Zuul green

Validated on shiftstackclient against live RHOSO + guest OCP 4.23 with collection from combine #34 (includes this PR through allowlist/OTE list fix 3259e02 + prior resolve/register commits).

Stage	Result
openstack_test	UNSTABLE with resolved counts (e.g. passed=20 failed=11); no `no tests where run!`; must-gather/post OK
lb_tests	Allowlist/OTE list OK (22 `[lb]` tests); run-test UNSTABLE; resolver e.g. passed=3 failed=4 skipped=15
egressip_tests	Pass — PLAY RECAP `failed=0 rescued=0`; post XML/HTML OK

Fixes covered in this PR that were exercised: OTE migration, result resolve, register-count wipe, allowlist quote/klog matching.

We will still run Zuul via testproject !2584 pinned to #34 for an official green before merge. Manual work was to de-risk; not a substitute for the check job.

tusharjadhav3302 requested review from chenwng, dlaw4608, ekuris-redhat, eshulman2, gryf, imatza-rh, mandre, stephenfin and winiciusallan as code owners August 7, 2026 11:48

tusharjadhav3302 added the ready-for-review PR is ready for code review label Aug 7, 2026

danchild reviewed Aug 7, 2026

View reviewed changes

Comment thread collection/stages/roles/openstack_test/tasks/run_openstack_test.yml Outdated

danchild reviewed Aug 7, 2026

View reviewed changes

Comment thread collection/stages/roles/openstack_test/tasks/run_openstack_test.yml

danchild reviewed Aug 7, 2026

View reviewed changes

Comment thread collection/stages/roles/openstack_test/tasks/run_openstack_test.yml Outdated

imatza-rh reviewed Aug 9, 2026

View reviewed changes

Comment thread collection/stages/roles/openstack_test/tasks/run_openstack_test.yml Outdated

openshift-ci Bot assigned imatza-rh Aug 9, 2026

openshift-ci Bot added the lgtm label Aug 9, 2026

tusharjadhav3302 force-pushed the adapt-openstack-test-ote branch from 2c5da98 to ec46f4e Compare August 9, 2026 16:04

openshift-ci Bot removed the lgtm label Aug 9, 2026

tusharjadhav3302 mentioned this pull request Aug 9, 2026

DNM: TP combine #32 + #33 #34

Open

2 tasks

tusharjadhav3302 added a commit that referenced this pull request Aug 10, 2026

Merge PR #33 OTE result-resolve fix into TP combine branch

d75b0e5

Keep OTE pass/fail counts when legacy count tasks are skipped

5781579

Skipped Ansible tasks still overwrite a shared register, which cleared OTE resolved counts and falsely triggered "no tests where run!". Co-authored-by: Cursor <cursoragent@cursor.com>

tusharjadhav3302 added a commit that referenced this pull request Aug 10, 2026

Merge PR #33 register-count fix into TP combine branch

954676c

tusharjadhav3302 mentioned this pull request Aug 10, 2026

Harden verification OpenStack CLI calls to use JSON output #32

Open

3 tasks

tusharjadhav3302 added a commit that referenced this pull request Aug 10, 2026

Merge PR #33 OTE allowlist/klog filter fix into TP combine

a65fb57

openshift-ci Bot added the lgtm label Aug 10, 2026

Uh oh!

Conversation

tusharjadhav3302 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

CLI / behavior trade-offs

Test plan

Links

Uh oh!

tusharjadhav3302 commented Aug 7, 2026

Manual validation evidence (serval70 / shiftstackclient)

1. Legacy path broken (root cause reproduced)

2. OTE build succeeds

3. List suite (replaces --dry-run)

4. Run one real test against the live cluster — PASSED

Notes

Uh oh!

danchild commented Aug 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tusharjadhav3302 commented Aug 8, 2026

Uh oh!

imatza-rh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

openshift-ci Bot commented Aug 9, 2026

Uh oh!

tusharjadhav3302 commented Aug 10, 2026

Follow-up: OTE false-failure counting (pushed)

Uh oh!

tusharjadhav3302 commented Aug 10, 2026

Uh oh!

tusharjadhav3302 commented Aug 10, 2026

Manual validation (serval71 / live RHOSO + guest OCP 4.23)

What we validated

Evidence (manual stages: ["openstack_test"])

Not claimed

Uh oh!

tusharjadhav3302 commented Aug 10, 2026

Manual validation (serval71 / live RHOSO + guest OCP 4.23)

What we validated

Evidence (manual stages: ["openstack_test"])

Not claimed

Uh oh!

tusharjadhav3302 commented Aug 10, 2026

Follow-up: lb_tests / egressip allowlist vs OTE list

Uh oh!

imatza-rh commented Aug 10, 2026

Uh oh!

tusharjadhav3302 commented Aug 10, 2026

Manual validation complete (serval71) — still targeting Zuul green

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

tusharjadhav3302 commented Aug 7, 2026 •

edited

Loading

3. List suite (replaces `--dry-run`)

Evidence (manual `stages: ["openstack_test"]`)

Evidence (manual `stages: ["openstack_test"]`)